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ABSTRACT 

Decision  makers  need  confidence  that  the  models  and  simulations  they  use  are  fit  to  support  their  decision 
making  process,  such  that  their  decisions  are  useful  to  their  specific  project  or  program.  It  is  entirely 
possible  and  even  quite  likely,  that  individual  models  and  simulations  or  those  expressly  for  federate  use 
are  already  available  somewhere  in  the  world.  Users  of  these  models,  simulations  or  federates  need  to 
appreciate  the  Validation  and  Verification  (V&V)  effort  already  applied  to  the  product,  and  to  understand 
the  level  of  impact  originally  anticipated  for  the  product  use.  This  effort  can  be  difficult  to  represent  and 
understand  when  use  and  re-use  are  required,  and  as  such,  time  and  further  effort  can  be  wasted  in 
duplicating  and  discussing  the  recorded  information  about  a  product  of  interest.  In  mitigation  to  this, 
a  standard  approach  to  recording  and  documenting  V&  V  information  is  introduced.  The  impact  level  of 
the  original  use  is  also  a  useful  data  item  in  understanding  the  V&V  effort  likely  to  have  been  exercised  on 
the  product.  This  information  is  usually  lost  or  not  even  recorded,  and  can  leave  subsequent  users  with 
some  uncertainty  about  the  suitability  of  the  product  for  their  purpose. 


1.0  INTRODUCTION 

The  high  importance  that  Modelling  and  Simulation  (M&S)  now  affords  National  and  International 
activities  is  becoming  increasingly  obvious.  The  implementation  of  M&S  activities  requires  the 
formulation  of  higher  levels  of  understanding  and  information  transfer  between  NATO,  the  PfP,  Industry 
and  National  organisations.  Verification  and  Validation  is  an  area  where  information,  data  and  the 
understanding  of  them  is  of  primary  importance.  Without  the  understanding,  the  information  is  nothing, 
and  the  goals  of  re-use  and  interoperability  are  not  possible. 

This  paper  is  organised  as  follows;  section  two  reviews  the  need  for  information,  understanding  and 
interoperability,  section  three  introduces  the  area  of  recording  verification  and  validation  information. 
The  new  International  Test  &  Operating  Procedure  (ITOP)  is  then  described  in  section  four  as  a  method 
for  standardising  the  information  from  the  V&V-related  M&S  activities.  The  methodologies  behind  the 
ITOP  are  discussed  in  section  five.  A  summary  entitled  6 1  don’t  care  what  you  do,  as  long  as  you  record  it 
consistently’  fills  section  six,  and  there  after  follows  the  conclusions  and  references. 


Paper  presented  at  the  RTO  NMSG  Conference  on  “NATO-PfP/Industry/National  Modelling 
and  Simulation  Partnerships  ”,  held  in  Paris,  France,  24-25  October  2002,  and  published  in  RTO-MP-094. 
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2.0  INFORMATION,  UNDERSTANDING  AND  INTEROPERABILITY 

Human  discourse  is  based  on  the  sharing  of  negotiated  representations,  and  as  such  the  management  and 
interoperability  of  models  and  simulations  is  grounded  in  communication  practice  [1].  These  statements 
were  made  concerning  the  risks  of  using  M&S  in  the  field  of  Agronomy  -  soil  management  and  crop 
production.  The  risks  detailed  are  of  surprising  relevance  to  the  military  domain  -  failure  to  appreciate 
structure  and  assumptions,  and  failure  to  appreciate  the  reliability  of  the  data  and  its  formulation  [1] 
to  mention  just  two.  The  offer  of  resolution  and  mitigation  is  stimulated  by  the  question  “What  kind  of 
representation  (of  information)  would  help?”  [1]. 

As  the  complexity  of  constructed  models,  and  we  would  claim,  their  predictive  power,  continues  to 
increase,  it  has  become  clear  that  efficient  and  informed  flow  of  information  between  modellers  will 
become  increasingly  important  and  even  vital  [2].  Again  this  is  a  statement  about  the  need  for  the 
management  of  information  about  models  and  simulations  -  this  time  from  the  field  of  Computational 
Neuroscience.  Following  some  research  into  what  neuro-biologists  felt  would  be  useful  to  them,  again  a 
surprisingly  relevant  set  of  requests  was  found  [2].  Of  most  relevance  to  this  particular  paper  were  ones 
concerning  the  importance  that  models  had  references  to  the  literature  and  information  on  the 
experimental  techniques  from  which  the  used  data  was  derived,  and  that  simulation  parameters  should  be 
retrievable  for  future  simulation  work  [2].  This  has  direct  implications  for  model  use,  re-use  and 
interoperability  in  our  field  as  well  as  the  above. 

Research  into  knowledge  management  models  [3]  has  shown  that  competitive,  commercial  and  potentially 
military  advantage  is  increased  through  the  interoperability  and  sharing  of  explicit  knowledge. 
In  particular  the  sharing  of  what  is  described  as  6 meta-knowledge’  about  reusability  is  described  as 
critical  for  competitive  advantage.  This  meta-knowledge,  I  would  suggest,  is  the  sort  of  information 
described  previously  in  [1]  and  [2]  -  experimental  techniques,  simulation  parameters,  data  formulation, 
model  structure  and  probably  most  importantly,  the  assumptions. 


3.0  RECORDING  VERIFICATION  AND  VALIDATION  INFORMATION 

The  concept  of  meta-knowledge  introduced  above  [3]  can  be  seen  to  represent  the  information  that  should 
be  documented  as  part  of  a  robust  verification  and  validation  procedure.  Every  little  piece  of  V&V  effort 
should  be  recorded  and  communicated  to  the  accreditation  authority,  such  that  they  may  be  convinced  of 
the  appropriateness  and  correctness  of  the  model  or  simulation  [4].  The  lack  of  this  type  of  information, 
alternatively  described  as  the  ‘complex  macro  statements’  has  been  quoted  in  a  number  of  studies  as 
reasons  for  the  unsuccessful  completion  of  studies  based  on  models  and  simulations  [5]. 

The  lack  of  a  standard  method  for  recording  the  meta-knowledge  or  macro  statements  about  the  V&V  of  a 
model  or  simulations  has  been  cited  a  number  of  times  as  the  reason  why  so  much  valuable  data  is  not 
available  for  use  and  exchange.  In  [6]  this  is  made  explicit  when  the  authors  state  that  ‘A  common  format 
for  V&V  reports  would  be  helpful  because  we  could  then  accumulate  this  type  of  information’.  They  go 
on  to  note  that  we  will  not  ‘soon  see  a  community  standard  for  information  items  or  formats’.  Even  within 
the  NATO  Modelling  and  Simulation  Group,  it  has  been  recognised  that  a  key  area  for  standardisation  is 
the  communication,  interaction  and  data  exchange  between  people  or  systems  [7]. 


4.0  THE  NEW  INTERNATIONAL  TEST  &  OPERATING  PROCEDURE  ON 
VERIFICATION  AND  VALIDATION  OF  MODELS  AND  SIMULATIONS 

Based  on  the  implementation  of  the  four-nation  Memorandum  of  Understanding  (FR,  GE,  UK,  U.S.) 
on  the  mutual  acceptance  of  test  and  evaluation,  the  International  Test  and  Evaluation  Steering  Committee 
(ITESC)  oversees  the  standardisation  and  documentation  of  test  operating  procedures  produced  by  specific 
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Working  Groups  of  Experts  (WGE).  The  work  of  the  committee  is  divided  into  eight  management  areas  of 
particular  concern  in  test  and  evaluation  of  military  materiel,  such  as  vehicles,  weapons/ammunition, 
aviation,  missiles  communications  /electronics,  etc.  In  1997  the  new  area  of  Modelling  and  Simulation 
(M&S)  was  introduced  and  a  Management  Committee  (MC7)  was  set  up  because  of  the  increasing 
importance  of  integrating  M&S  into  other  ITOP  areas.  The  role  of  MC7  is  to  provide  a  co-ordinated 
approach  to  the  subject  among  its  WGE  and  the  ITESC  and  to  the  WGE  of  the  other  management  areas. 

Working  group  7.2  has  been  focussing  on  the  use  of  verification  and  validation,  its  main  activities  have 
been: 

•  Preparing  procedure  and  guidance  documentation  on  the  optimum  use  of  V&V  for  other  WGE; 

•  Preparing  procedure  and  guidance  information  on  how  to  transfer  information  from  the  V&V 
process  to  other  nations; 

•  Promoting  the  use  of  defined  V&V  frameworks  in  T&E; 

•  Assisting  other  WGE  in  their  use  of  V&V  as  regards  their  own  simulations; 

•  Reviewing  research  developments  into  methods  and  tools  useable  in  V&V  and  to  facilitate  their 
adoption  where  applicable; 

•  Build  a  reliable  basis  for  future  Accreditation  of  models. 

This  has  resulted  in  the  production  of  an  ITOP  document  which  can  be  utilised  across  the  M&S  and  V&V 
community,  if  the  community  wants  it. 


5.0  CONCEPTS  BEHIND  THE  V&V  ITOP  DOCUMENT 

This  ITOP  applies  to  the  V&V  activities  associated  with  models,  data,  and  model  use  (or,  more  correctly 
its  simulation)  which  are  intended  to  support  primarily  defence  applications,  particularly  where  the  mutual 
acceptance  of  results  and  information  derived  from  the  M&S  is  a  key  consideration  for  the  reciprocal 
procurement  of  defence  equipment.  This  section  introduces  key  concepts  that  are  used  in  the  ITOP  [8], 
these  are  a  “V&V  cases”  concept,  a  “claim-argument-evidence”  structure,  and  a  “levels”  concept  for  the 
classification  of  M&S-use  impact  and  V&V  activities. 

To  promote  the  avoidance  of  unnecessary  re-analysis  and  evaluation,  achieved  V&V  information  from 
three  elements  (data,  model,  and  simulation)  shall  be  documented  in  three  separate  cases.  This  separation 
is  done  because  it  was  felt  that  these  were  the  information  ‘blocks’  which  were  most  likely  to  be 
transferred  and  re-used. 

The  concept  of  a  claim  argument  evidence  structure  arose  from  the  need  to  record  the  justification  and 
reasoning  behind  important  decisions.  The  precise  way  in  which  a  claim  for  accreditation  is  divided  into 
multiple  lower  claims  has  to  be  explicit  and  traceable  -  this  is  called  the  argument.  The  sub-claims  may 
also  be  divided  in  to  further  claims  by  further  argument  until  a  hierarchy  of  claim  and  argument  has  been 
constructed.  Eventually,  the  lower  level  claims  should  be  able  to  be  substantiated  by  a  piece  of  evidence 
obtained  from  the  V&V  effort. 

A  levels  concept  assists  in  communication  and  understanding  between  parties  in  discussion.  It  also 
provides  a  convenient  metric  for  comparison  purposes  ensuring  that  there  is  some  consistency  and 
standardisation  between  the  entities  of  interest  and  some  frame  of  reference^].  In  the  new  procedure  there 
are  two  types  of  level  of  consequence  -  the  impact  level  that  the  M&S  is  to  be  used  for  (e.g.  a  level  of 
commercial,  project,  or  human  impact  that  would  be  experienced  from  the  mis-use  of  the  product);  and  the 
level  of  effort  (and  therefore  cost)  required  to  undertake  verification  and  validation  activities  in  order  to 
generate  a  requisite  amount  of  confidence  that  the  model,  data,  or  simulation  correctly  satisfies  its 
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purpose.  There  are  obvious  links  between  these  two.  A  simulation  used  to  predict  kill/survivability  rates 
would  probably  score  a  high  impact  level.  As  such  it  would  likewise  require  a  high  level  of  V&V  effort  to 
provide  evidence  that  it  did  behave  appropriately. 


6.0  I  DON’T  CARE  WHAT  YOU  DO  AS  LONG  AS  YOU  RECORD  IT 
CONSISTENTLY 

Documentation  should  be  a  normal  part  of  modelling  and  simulation,  for  example  some  modem 
simulations  do  include  their  documentation  on-line  within  the  code  [6].  This  is  a  good  situation  as  the 
information  is  likely  to  be  current,  controlled  and  will  actually  take  up  very  little  of  a  computer’s  storage 
capacity.  A  standardised  or  consistent  format  for  this  type  of  information  electronic  or  hardcopy  would  be 
very  helpful  because  the  information  would  become  easier  to  identify  and  could  even  be  automatically 
read  and  accumulated  [6]. 

The  need  for  some  generic  document  referred  to  as  a  Togbook’  has  been  identified  as  very  useful  [5], 
particularly  for  recording  assumptions  about  the  data,  the  model,  the  simulation  mn,  or  even  just  about  the 
state  of  the  real  world.  A  document  like  this  would  have  a  number  of  potentially  very  important  uses  e.g.: 

•  An  audit  chain  for  decision  making 

•  A  stmctured  walkthrough  of  the  conceptual  model  [6] 

•  An  assessment  document  for  other  applications 

•  An  item  of  evidence  to  justify  accreditation 

The  new  ITOP  aims  to  satisfy  these  requirements  and  uses  by  providing  a  template  for  such  a  document; 
a  rationale  for  its  use  and  some  examples  of  how  to  apply  the  concepts  discussed. 


7.0  CONCLUSIONS 

A  new  international  standard  for  the  recording  of  verification  and  validation  effort  has  been  constructed. 
It  introduces  several  new  concepts  to  this  area  including  levels  and  the  construction  of  reasoned  arguments 
for  accreditation.  It  is  likely  to  have  a  dramatic  impact  on  improving  interoperability  and  communication 
between  services,  industry  and  nations.  It  will  satisfy  a  requirement  in  the  field  of  modelling  and 
simulation  for  standardisation  in  this  area. 

Open  distribution  of  this  and  any  other  ITOP  is  limited  to  FR/GE/UK/US  Government  agencies  only. 
Requests  from  other  countries  should  be  referred  to  U.S.  Army  Developmental  Test  Command,  ATTN: 
CSTE-DTC-TT-M  Aberdeen  Proving  Ground,  MD  21005-5055.  In  all  cases  the  ITOP  reference  number 
and  title  should  be  quoted.  Anyone  can  obtain  a  copy  of  an  ITOP  as  long  as  there  is  no  objection  from  the 
four  countries. 
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The  requirement  for  M&S  V&V 
information  management 

“There  is  a  failure  to  appreciate  model  structure  and 
assumptions,  and  a  failure  to  appreciate  the  reliability  of  the 
data  and  its  formulation” 

P.  G.  Cox:  “The  management  of  models:  risks  and  responsibility.”  Proceedings  of  the  9th  Australian 

Agronomy  Conference  1998 

“It  is  important  that  models  have  references  to  the  literature 
and  information  on  the  experimental  techniques  from  which 
the  used  data  was  derived,  and  that  simulation  parameters 
should  be  retrievable  for  future  simulation  work” 

S.  Koslow  &  M.  Huerta  (Eds):  “Progress  in  Neuroinformatics.”  Laurence  Erlbaum  Associates  1996. 
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The  requirement  for  M&S  V&V 
information  management 

“A  common  format  for  V&V  reports  would  be  helpful 
because  we  could  then  accumulate  meta-knowledge  or 
macro  statements  about  the  M&S  V&V” 

D.J  Cloud  &  L.B  Rainey  (eds):  “Applied  Modeling  and  Simulation  :  An  integrated  approach  to  development 

and  operation.”  McGraw-Hill  International  Inc.  1998. 

“A  key  area  for  standardisation  is  the  communication, 
interaction  and  data  exchange  between  people  or  systems  ” 

E.  Schwan:  “The  NATO  Modelling  and  Simulation  Group  (NMSG)  and  the  Role  of  M&S  Standards.” 

Proceedings  of  the  NSMG  Conference  Breda,  November  2001 . 
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International  Test  &  Evaluation  Steering  Committee 

and  Management  Committee  7 

(Modelling  &  Simulation) 


Based  on  the  implementation  of  the  four-  nation 
Memorandum  of  Understanding  on  the  mutual 
acceptance  of  test  and  evaluation,  the  International  Test 
and  Evaluation  Steering  Committee  (ITESC)  oversees 
the  standardisation  and  documentation  of  test  operating 
procedures  produced  by  specific  Working  Groups  of 
Experts 

The  WGE  are  made  up  of  experts  in  the  particular  field 
from  all  four  member  nations,  who  work  together  to 
produce  International  Test  Operation  Procedures  Qjne^Q 
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The  role  of  an  ITOP  1 

*  Some  of  the  many  benefits  include: 

data  can  be  reliably  reused; 

>  International  co-operation  &  competition  becomes 
much  easier  and  more  effective; 

duplication  of  testing  is  avoided; 

>There  is  confidence  in  components  and  systems 
tested ; 

*  ITOPs  provide  a  convenient  basis  for  defining  test 
procedures  in  international  contracts 
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The  role  of  an  HOP  1 

•  The  work  of  the  ITESC  and  the  ITOPs  Working 
Groups  provide  a  valuable  basis  for  these 
developments. 

•  In  1997  the  new  area  of  Modelling  and  Simulation  was 
introduced  and  a  Management  Committee  (MC7)  was 
set  up. 

•  Many  ITOPs  have  been  used  as  the  basis  for  standard 
agreements  (STANAGS)  across  the  NATO 
community,  leading  to  the  larger  scale  standardisation 
of  T&E  practises. 
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WGE  7.2  Verification  &  Validation 

Activities  include: 

>  Preparing  procedure  and  guidance  documentation  on 
the  optimum  use  of  V&V  for  other  WGE; 

>  Preparing  procedure  and  guidance  information  on  how 
to  transfer  information  from  the  V&V  process  to  other 
nations; 

> Promoting  the  use  of  defined  V&V  frameworks  in  T&E; 

>  Reviewing  research  developments  into  methods  and 
tools  useable  in  V&V  and  to  facilitate  their  adoption 
where  applicable; 

>  Build  a  reliable  basis  for  future  Accreditation  of  models. 
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Concepts  behind  the  V&V  ITOP  document 

•  This  ITOP  applies  to  the  V&V  activities  associated  with 

,  ,  and  which  are  intended  to 

support  primarily  defence  applications. 

•  These  are  the  information  ‘blocks’  which  are  most 
likely  to  be  transferred  and  re-used. 

•  Achieved  V&V  information  from  three  elements  shall 
be  documented  in  three  separate 


QinetiQ 


13-9 


Concepts  behind  the  V&V  ITOP  document 

•  The  concept  of  a  claim  argument  evidence  structure 
arose  from  the  need  to  record  the  justification  and 
reasoning  behind  important  decisions. 

•  The  precise  way  in  which  a  claim  for  accreditation  is 
divided  into  multiple  lower  claims  has  to  be  explicit  and 
traceable  -  this  is  called  the  argument. 

•  Eventually,  the  lower  level  claims  should  be  able  to  be 
substantiated  by  a  piece  of  evidence  obtained  from  the 
V&V  effort. 
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Concepts  behind  the  V&V  ITOP  document 

•  A  levels  concept  assists  in  communication  and 
understanding  between  parties  in  discussion. 

•  In  the  new  ITOP  procedure  there  are  types  of  level 

>  The  that  the  M&S  is  to  be  used  for. 

(e.g.  a  level  of  commercial,  project,  or  human  impact  that 
would  be  experienced  from  the  mis-use  of  the  product) 

>  The  required  to  undertake  verification  and 

validation  activities  in  order  to  generate  a  requisite 
amount  of  confidence. 
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Recording  the  V&V  information 

•  The  need  for  some  generic  document  referred  to  as  a 
‘  ’  has  been  identified  as  very  useful, 

particularly  for  recording  assumptions  about  the  data, 
the  model,  the  simulation  and  even  the  assumed  ‘real 
world’.  A  document  like  this  would  have  a  number  of 
potentially  very  important  uses  e.g. 

•  An  audit  chain  for  decision  making 

•  A  structured  walkthrough  of  the  conceptual  model 

•  An  assessment  document  for  other  applications 

•  An  item  of  evidence  to  justify  accreditation 
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Recording  the  V&V  information 

•  The  new  ITOP  aims  to  satisfy  these  requirements  and 
uses  by  providing 

-  A  template  for  such  a  document; 

-  A  rationale  for  its  use; 

-  Some  examples  of  how  to  apply  the  concepts. 
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Obtaining  ITOP  information 

•  Open  distribution  of  this  and  any  other  ITOP  is  limited 
to  FR/GE/UK/US  Government  agencies  only. 

•  Requests  from  other  countries  should  be  referred  to 
U.S.  Army  Developmental  Test  Command,  ATTN: 
CSTE-DTC-TT-M  Aberdeen  Proving  Ground,  MD 
21005-5055.  In  all  cases  the  ITOP  reference  number 
and  title  should  be  quoted. 

•  The  ITOP  on  V&V  reference  number  is 

•  Anyone  can  obtain  a  copy  of  an  ITOP  as  long  as  there 
is  no  objection  from  the  four  countries. 
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Thank  you 


Mr  Richard  Lawrence  Maguire 
QinetiQ  -  MoD  Boscombe  Down 
Tel:  +44  1980  662292 
Email:  rlmaquire@qinetiq.com 
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